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ABSTRACT 


The  U.S.  military  currently  uses  a  number  of  virtual  marksmanship  simulators  to  help 
train  its  soldiers  and  Marines  in  marksmanship  techniques.  Like  all  information  systems, 
these  virtual  marksmanship  simulators  rapidly  become  outdated,  and  need  to  be  upgraded 
or  replaced.  One  aspect  that  can  potentially  be  improved  is  the  “realism”  of  these 
simulators,  generally  in  regards  to  their  graphics  quality.  The  issue  is  that  simulator 
realism  is  a  highly  complex  subject,  with  many  more  factors  to  account  for  besides  just 
graphics  quality.  The  author  hypothesized  that  (1)  one  or  more  specific  factors  can  be 
identified  which  impact  the  realism  of  virtual  marksmanship  simulators,  and  (2) 
improving  or  enhancing  a  factor,  such  as  graphics  quality,  will  have  a  positive  effect  on  a 
participants  objective  score  and/or  their  subjective  experience  with  the  simulator. 

A  group  of  17  civilian  and  active-duty  military  volunteers  conducted  baseline 
training  with  the  Indoor  Simulated  Marksmanship  Trainer  (ISMT),  and  then  participated 
in  two  different  scenarios  using  the  standard  Navy  Handgun  Qualification  Course 
(NHQC).  One  scenario  used  a  low-resolution  background,  while  the  other  used  a  high- 
resolution  background. 

Results  partially  supported  the  hypotheses.  While  no  conclusive  objective  data 
identified  any  specific  factors  that  impact  the  realism  of  virtual  marksmanship  simulators, 
there  was  subjective  data  to  suggest  that  graphics  quality  did  impact  the  realism  of  virtual 
marksmanship  simulators.  Likewise,  the  objective  data  taken  from  shooters’  scores 
showed  no  improvement  between  low-resolution  and  high-resolution  scenarios,  but  the 
subjective  experience  of  the  participants  trended  positively  toward  higher  fidelity 
graphics. 
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I.  INTRODUCTION 


A.  OVERVIEW 

The  U.S.  military  currently  uses  a  number  of  virtual  marksmanship  simulators  to 
help  train  its  Soldiers  and  Marines  in  marksmanship  techniques.  Like  all  information 
systems,  these  virtual  marksmanship  simulators  rapidly  become  outdated,  and  need  to  be 
upgraded  or  replaced.  One  aspect  that  can  potentially  be  improved  is  the  “realism”  of 
these  simulators,  generally  in  regards  to  their  graphics  quality.  The  issue  is  that  simulator 
realism  is  a  highly  complex  subject,  with  many  more  factors  to  account  for  besides  just 
graphics  quality.  Of  course,  the  critical  issue  is  not  simulator  fidelity  at  all  but  rather 
trainee  perfonnance  or  training  transfer  of  skilled  marksmanship  performance.  Thus,  this 
thesis  will  investigate  this  issue  from  both  the  technology  and  performance  viewpoints. 

B.  PROBLEM  STATEMENT 

The  U.S.  military  is  seeking  to  improve  or  replace  a  number  of  aging  virtual 
marksmanship  simulators  with  newer,  more  realistic  systems.  However,  it  is  unclear  what 
factors  impact  the  “realism”  of  these  simulators.  This  limits  our  ability  to  improve  these 
simulators,  which  in  turn  limits  our  ability  to  enhance  their  training  potential.  The 
problem  is  that  the  relationship  between  simulator  fidelity  and  efficacy  of  the  training 
system  is  unknown. 

C.  PURPOSE  STATEMENT 

The  purpose  of  this  research  is  to  determine  what  factors  impact  the  realism  of 
virtual  marksmanship  simulators,  identify  how  these  factors  can  be  improved,  and 
measure  the  resulting  impact  on  trained  perfonnance  This  will  allow  us  to  enhance  the 
realism  of  these  factors,  which  will  in  turn  enhance  the  overall  realism  of  the  simulators. 
This  is  important  because  being  able  to  clearly  define  what  makes  these  simulators  more 
realistic  will  lead  to  further  research  to  determine  if  simulators  with  enhanced  realism  are 
more  effective  at  training  than  their  less  realistic  counterparts. 
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D.  RESEARCH  QUESTIONS  AND  HYPOTHESES 


1.  Question  One 

Are  there  specific  factors  which  impact  the  realism  of  virtual  marksmanship 
simulators? 


HOi:  No  specific  factors  can  be  identified  which  impact  the  realism  of  virtual 
marksmanship  simulators. 

HAi:  One  or  more  specific  factors  are  identified  which  impact  the  realism  of 
virtual  marksmanship  simulators. 

2.  Question  Two 

Does  improving  or  enhancing  these  factors  enhance  the  overall  simulation? 

HO2:  Improving  or  enhancing  a  factor,  such  as  graphics  quality,  will  have  no 
discernible  effect  on  a  participant’s  objective  score  or  their  subjective  experience  with  the 
simulator. 


HA2:  Improving  or  enhancing  a  factor,  such  as  graphics  quality,  will  have  a 
positive  effect  on  a  participant’s  objective  score  and/or  their  subjective  experience  with 
the  simulator. 

E.  ORGANIZATION  OF  THE  THESIS 

This  thesis  is  organized  in  the  following  chapters: 

•  Chapter  I:  Introduction.  This  chapter  gives  a  general  outline  of  the  work 
and  defines  the  problem  addressed  by  this  research. 

•  Chapter  II:  Background.  This  chapter  discusses  existing  research  on 
simulator  realism,  marksmanship,  and  the  use  of  virtual  marksmanship 
simulators. 

•  Chapter  III:  Method.  This  chapter  discusses  in  detail  the  design  of  the 
experiment  and  the  collection  of  data  for  the  thesis. 

•  Chapter  IV:  Results.  This  chapter  discusses  the  data  collected  from  the 
experiments  and  the  analysis  of  it. 
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Chapter  V:  Conclusions.  This  chapter  describes  the  conclusions  drawn 
from  the  results  of  the  experiments,  and  offers  some  ideas  for  potential 
future  research. 
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II.  BACKGROUND 


A.  INTRODUCTION 

While  no  specific  research  has  been  done  to  detennine  what  factors  determine 
realism  in  virtual  marksmanship  simulators,  there  have  been  numerous  other  studies 
which  look  at  realism  in  different  contexts,  both  in  regards  to  simulators  and  otherwise. 
There  have  also  been  many  texts  written  on  the  topic  of  marksmanship  training.  Lastly 
there  is  a  growing  body  of  research  on  the  effective  use  of  simulators  and  virtual 
environments  for  training.  A  discussion  of  these  three  areas  and  their  overlapping 
concern  follows  in  the  form  of  a  literary  review. 

Additional  background  information  will  come  from  Meggitt  Training  Systems 
(n.d.),  makers  of  the  Indoor  Simulated  Marksmanship  Trainer  (ISMT)  used  by  the  United 
States  Marine  Corps,  and  from  Cubic  Defense  Systems  (n.d.),  makers  of  the  Engagement 
Skills  Trainer  2000  (EST  2000)  used  by  the  United  States  Army.  A  thorough  review  of 
these  systems  will  be  conducted  to  learn  their  capabilities  and  limitations,  and  determine 
what,  if  any  features  can  be  improved  to  enhance  the  overall  realism  of  the  simulators. 

B.  MARKSMANSHIP  TRAINING 

Little  is  known  about  how  ISMT  training  transfers  to  live  fire.  LTC  Yates,  United 
States  Marine  Corps  (USMC),  studied  the  effectiveness  of  the  ISMT  to  train  Marines  in 
marksmanship  fundamentals  (2004).  Using  28  participants  at  Marine  Corps  Recruit 
Depot,  San  Diego,  during  the  initial  rifle  qualification  using  the  USMC  course  of  fire,  a 
side-by-side  comparison  was  conducted  to  measure  the  performance  of  Marines  trained 
in  ISMT  compared  to  Marines  who  were  trained  using  dry  fire,  culminating  in  a  final 
live-fire  qualification.  The  results  showed  no  significant  difference  in  group  shots  and 
scores;  in  other  words,  the  Marines  who  trained  with  ISMT  performed  as  well  as  those 
who  trained  without  ISMT.  LTC  Yates  concluded  that  there  was  a  lack  of  evidence  to 
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support  ISMT  being  used  as  the  only  training  mechanism  without  also  providing  expert 
instruction.  The  importance  of  this  study  was  that  the  ISMT  training  was  as  effective  as 
dry  fire. 

The  U.S.  Army  Research  Laboratory  conducted  a  training  transfer  study,  A 
Comparison  of  Live  and  Simulated  Fire  Soldier  Shooting  Performance  (Scribner,  2007), 
using  the  Dismounted  Infantry  Survivability  and  Lethality  Testbed  simulation-based 
training  to  determine  the  transfer  to  live  training.  12  participants  completed  a  course  of 
fire  using  the  M16  to  fire  at  18  pop-up  target  silhouettes.  Firing  ranges  consisted  of  75, 
100,  150,  200,  250,  and  300-meter  targets  from  a  kneeling,  foxhole-supported  position. 
Each  participant  was  exposed  to  10  trials  of  simulation  and  live  fire,  and  hit  percentage, 
shot-reaction  time,  shooting  performance,  and  radial-aiming  error  were  assessed  under 
both  conditions.  Both  simulation  and  live  fire  showed  similar  hit  percentages,  indicating 
that  participants  maintained  performance  from  simulation  to  live  fire.  In  live  fire,  each 
shot  took  less  time  and  reaction  times  were  faster  compared  to  the  shots  and  reaction 
times  in  the  simulation.  The  difference  in  reaction  times  may  be  attributed  to  many 
factors  including  human  attributes  such  as  rifle  movement  with  a  live  weapon  or  outside 
weather  conditions.  The  notion  that  there  is  a  strong  relationship  between  basic 
marksmanship  skills  in  simulation  and  live  fire  is  supported  by  this  research. 

The  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences’  Research 
Report  1761  (Hagman,  2000)  compared  simulated  marksmanship  training  to  standard 
Army  instruction  with  a  final  live  fire  qualification  course  of  fire.  184  participants  trained 
with  simulation,  while  202  participants  trained  with  standard  Army  instruction.  All 
participants  completed  1 1  periods  of  instruction,  ranging  from  introduction  to  basic  rifle 
marksmanship  and  mechanical  training  to  practice  firing,  leading  up  to  firing  for 
qualification.  The  dependent  variables  were  the  number  of  targets  hit  and  rounds  fired 
during  training  and  qualification  fire.  Results  indicated  that  the  use  of  simulation  for 
basic  rifle  marksmanship  training  reduced  the  number  of  live  rounds  fired,  increased  the 
number  of  participants  firing  to  standard  requirements,  and  increased  the  number  of 
target  hits.  Simulation  did  not  improve  record  fire  qualification  scores.  The  results  of  the 
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Research  Report  1761  (Hagman,  2000)  suggest  that  simulation-based  marksmanship 
training  would  improve  initial  entry  rifle  marksmanship  performance,  while  saving 
ammunition. 

Getty  (2008)  compared  simulation  marksmanship  training  using  the  ISMT  to 
SNMT  with  a  one-week  gap  between  training  and  qualification  course  of  fire.  A  between- 
groups  study  was  conducted  with  random  selection  of  participants  blocked  by  previous 
marksmanship  experience  in  which  half  of  the  participants  received  training  in  the  ISMT, 
while  the  other  half  received  SNMT.  The  dependent  variables  were  diameter  size  of 
average  group  shots,  diameter  size  of  group  shots  and  length  of  mean  point  of  impact 
(MPI)  to  center  zeroing  point  for  the  three,  seven,  and  15-yard  lines,  and  overall  scores 
on  the  NHQC  course  of  fire.  Getty’s  findings  show  there  was  no  change  in  participants’ 
marksmanship  perfonnance  and  scores  regardless  of  whether  they  received  the  ISMT 
training  or  the  SNMT,  except  for  the  seven-yard  line.  For  the  seven-yard  line,  participants 
who  received  the  ISMT  training  showed  greater  improvements  in  performance  than 
participants  who  received  SNMT.  Exploratory  analyses  suggest  that  simulation  training 
may  be  most  beneficial  for  less  experienced  shooters.  Additionally,  there  was  no 
significant  difference  in  skill  retention  after  one  week  between  the  ISMT  participants  and 
SNMT  participants,  suggesting  that  a  longer  period  of  time  is  needed  to  detect  group 
differences  in  the  retention  of  skills. 

In  summary,  the  studies  by  LTC  Yates  (2004),  ARL-TR-4234  (Scribner,  2007), 
and  Research  Report  1761  (Hagman,  2000)  show  that  simulation  is  at  least  as  effective  in 
marksmanship  training  as  current  standard  Army  and  Marine  marksmanship  training  and 
can  be  a  cost-effective  way  to  train.  However,  neither  of  these  two  key  reports  identified 
the  roll  of  fidelity  to  training  effectiveness. 

C.  SIMULATORS  AND  VIRTUAL  ENVIRONMENTS 

Despite  how  commonly  virtual  environment  trainers  are  used  there  are  many 
fielded  virtual  environment  trainers  for  which  there  has  been  no  detailed  study  conducted 
to  validate  the  effectiveness  of  a  virtual  environment  trainer.  Such  studies  are  referred  to 
as  verification  of  skills  acquisition  (or  training  transfer.)  A  positive  verification  of  skill 
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acquisition  requires  a  quantified  measure  of  improvement  at  task  perfonnance 
(Fredriksen  &  White,  1989).  To  justify  the  expense  of  developing  and  fielding  virtual 
environment  trainers  they  must  be  verified  to  accomplish  skill  acquisition  as  well  as 
conventional  methods  of  training,  or  a  reduced  level  of  effectiveness  must  be  accepted  as 
a  trade-off  for  reduced  cost  or  increased  safety. 

It  is  a  common  misunderstanding  by  some  potential  consumers  of  virtual 
environment  trainer  technology  that  a  virtual  environment  trainer  is  a  revolutionary 
device  that  must  or  should  accomplish  training  transfer  more  rapidly  than  conventional 
training  methods.  Since  Edward  Thorndike  published  his  seminal  work  on  educational 
research  over  90  years  ago,  there  has  been  debate  about  whether  a  certain  type  of  media 
used  in  training  or  education  is  qualitatively  superior  to  other  methods  (Thorndike,  1912). 
A  virtual  environment  trainer  can  be  viewed  as  simply  a  different  media  of  instruction 
(Clark,  1983).  Some  studies  have  shown  a  difference  in  relative  achievement  between 
groups  that  received  training  on  the  same  topic  using  different  media;  e.g.,  written 
instruction  compared  to  graphical  instruction  (Reiser  &  Gagne,  1982). 

Other  studies  have  suggested  that  the  learning  has  little  to  do  with  the  media  used 
and  more  to  do  with  the  methods  of  instruction  (Clark,  1983).  That  is  to  say  that  a  three 
dimensional  training  virtual  enviromnent  might  not  be  a  superior  tool  for  training  to  a  less 
sophisticated  analog  of  the  task.  While  the  debate  over  the  relevance  of  media  in  training 
has  not  been  resolved,  it  is  likely  that  the  potential  for  a  relationship  between  media  and 
learning  should  be  viewed  as  an  interaction  between  cognitive  processes  and 
characteristics  of  the  external  world.  An  effective  trainer  must  replicate  the  actual  task  to 
the  extent  that  it  results  in  the  same  cognitive,  perceptual  and  motor  processes  as  the 
actual  task.  A  device  that  achieves  rapid  training  or  skills  development  is  certainly  worth 
pursuing  but  the  potential  benefits  of  most  existing  virtual  environment  trainers  lie  in  the 
ability  to  conduct  training  inexpensively,  safely,  and  at  the  times  and  places  of 
convenience  rather  than  tying  training  to  a  geographic  location.  A  virtual  environment 
trainer  may  not  be  a  revolutionary  device  but  it  can  fill  a  niche  that  conventional  training 
methods  cannot  (Schlager,  1994). 
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This  thesis  is  concerned  specifically  with  validating  the  effectiveness  of  the  ISMT 
(Indoor  Simulated  Marksmanship  Trainer)  at  training  novice  shooters  in  the  fundamentals 
of  marksmanship.  The  ISMT  is  a  virtual  environment  trainer  that  employs  modified 
infantry  weapons,  a  video  projection  system,  and  a  precision  sensor  controlled  by  two 
Microsoft  Windows  based  computers.  The  ISMT  was  not  the  first  virtual  environment 
trainer  developed  for  marksmanship  training.  In  the  early  1980s  the  U.S.  Army  developed 
the  Weaponeer  marksmanship  trainer  and  employed  it  to  teach  initial  entry  soldiers  the 
fundamentals  of  marksmanship  (Schendel  &  Heller,  1985).  The  Anny  Research  Institute 
developed  the  Multipurpose  Arcade  Combat  Simulator  (MACS)  as  a  trainer  for  basic 
marksmanship  at  about  the  same  time  that  Weaponeer  was  developed  (White,  Carson,  & 
Wilbourn,  1991).  Having  evaluated  these  systems  the  Marine  Corps  developed  a 
requirements  document  for  a  marksmanship  trainer  for  which  the  ISMT  was  eventually 
procured. 

D.  REALISM 

While  no  specific  research  has  been  done  to  detennine  what  factors  determine 
realism  in  virtual  marksmanship  simulators,  there  have  been  numerous  other  studies 
which  look  at  realism  in  different  contexts.  Fencott  (1999)  believes  that  as  presence  is 
based  on  perception,  it  is  the  content  of  the  virtual  environment,  and  how  that  content  is 
designed,  that  is  most  important.  Prothero,  Parker,  Furness  and  Wells  (1995)  agree  with 
this,  saying  that  presence  is  enhanced  by  how  the  user  perceives  the  space,  specifically, 
an  increased  sense  of  presence  results  from  a  wide  field  of  view  and  a  sense  of 
foreground  and  background,  which  enables  the  user  to  orient  themselves  in  space  and 
understand  the  orientation  of  virtual  objects  in  the  same  space.  McMahan  (2003) 
describes  realism  as  how  accurately  the  virtual  environment  represents  objects,  events 
and  people.  She  goes  on  to  say  realism  is  subdivided  into  social  realism  (the  extent  to 
which  the  social  interactions  in  the  virtual  reality  environment  matched  interactions  in  the 
real  world),  and  perceptual  realism  (how  closely  do  the  objects,  environments,  and  events 
depicted  match  those  that  actually  exist).  In  looking  at  medical  simulators,  Alessi  and 
Trollip  (2001)  introduce  a  model  which  suggests  that  as  realism  increases  then  the 
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learning  outcomes  reach  an  optimum  level  but  subsequently  will  drop  off  for  individuals 
who  are  novices  or  of  intermediate  knowledge  in  the  skills  or  content  domain  of  the 
simulation  or  educational  game.  Research  done  by  Tashiro  and  Dunlap  (2007)  explains 
that  this  decrease  in  learning  as  realism  increases  could  be  due  to  a  confounding  of 
realism  and  engagement  with  less  motivation  to  remain  engaged  as  realism  swamps  the 
novice  and  intermediate  knowledge  users  within  the  simulation  or  gaming  environment. 
Brooks  (1999)  describes  a  number  of  shortcomings  in  virtual  reality  simulators,  and 
offers  potential  ways  to  improve  them. 
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III.  METHOD 


A.  OVERVIEW  OF  RESEARCH  DESIGN 

The  experimental  design  for  this  thesis  utilized  a  within-groups  study  of  civilian 
and  active  duty  military  volunteers  randomly  selected  to  complete  two  different  scenarios 
in  the  ISMT.  Marksmanship  performance  was  measured  by  score  using  the  NHQC  firing 
sequence.  This  thesis  research  was  approved  by  the  Naval  Postgraduate  School  (NPS) 
Institutional  Review  Board  (IRB);  IRB  approval  number  NPS.2012.0080-IR-EP7-A. 

B.  PARTICIPANTS 

All  participants  were  either  students  or  faculty  at  NPS.  A  total  of  17  participants, 
including  15  active  duty  military  and  two  civilians,  took  part  in  this  study.  Of  the  15 
active  duty  military  participants,  two  were  from  foreign  services,  and  the  remaining  13 
were  from  the  U.S.  military.  Of  the  17  initial  participants,  all  17  completed  the  pre¬ 
experiment  demographics  survey.  Table  1  characterizes  the  participants’  general 
demographics,  while  Table  2  characterizes  their  previous  marksmanship  experience. 
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Table  1.  General  Statistics  from  Participants’  Demographic  Surveys 


Demographics  Survey  (General) 

Age  (%):  Years 

28-32 

17.65% 

33-37 

41.18% 

38-42 

23.53% 

43+ 

17.65% 

Corrected  Vision  (%): 

Yes 

41.18% 

No 

58.82% 

Branch  of  Service  (%): 

Navy 

52.94% 

Marines 

17.65% 

Army 

11.76% 

Air  Force 

5.88% 

Civilian 

11.76% 

Job  Specialty  (%): 

Combat  Element 

35.29% 

Support  Element 

64.71% 

Video  Game  Player  (%): 

Yes 

47.06% 

No 

52.94% 

Video  Game  Play  Time  (%):  Hours  per  Week 

0-5 

76.47% 

6-10 

17.65% 

11-15 

5.88% 

16-20 

0.00% 

21+ 

0.00% 

The  general  infonnation  from  the  demographic  surveys  was  collected  and 
analyzed  in  order  to  determine  if  there  were  any  other  scientifically  interesting  elements 
which  may  impact  the  participants’  scores.  These  findings  are  discussed  in  further  detail 
in  Chapter  IV. 
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Table  2.  Marksmanship  Specific  Statistics  from  Participants’  Demographic  Survey 


Demographics  Survey  (Marksmanship) 

Qualified  With  M9  (%): 

Yes 

88.24% 

No 

11.76% 

Self-Reported  Currency  (%): 

Very  Current 

5.88% 

Current 

11.76% 

Neutral 

17.65% 

Out  of  Date 

47.06% 

Very  Out  of  Date 

17.65% 

Self-Reported  Proficiency  (%): 

Very  Proficient 

11.76% 

Proficient 

17.65% 

Average 

47.06% 

Beginner 

5.88% 

Complete  Novice 

17.65% 

Used  Virtual  Marksmanship  Simulators  (%): 

Yes 

52.94% 

No 

47.06% 

The  marksmanship  information  from  the  demographic  surveys  was  collected  and 
analyzed  in  order  to  determine  if  participants’  prior  marksmanship  experience  had  an 
effect  on  their  scores  in  the  testing.  These  findings  are  discussed  in  further  detail  in 
Chapter  IV. 

C.  EQUIPMENT 

1.  ISMT 

ISMT  is  currently  employed  by  the  USMC  as  a  portable,  stand-alone 
marksmanship  trainer.  ISMT  is  a  “three  dimensional  simulation  based  trainer  for  indoor 
use,  capable  of  instructing  in  basic  and  advanced  marksmanship,  shoot/no-shoot 
judgment,  combat  marksmanship,  and  weapons  employment  tactics”  (U.S.  Marine  Corps 
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Concepts  in  Programs,  2008,  p.  214).  ISMT  has  the  capability  to  use  a  wide  variety  of 
weapons,  including  the  .50cal.  machinegun  (M2),  9  mm  Beretta  (M9),  5.56  mm  service 
rifle  (Ml 6),  7.62  mm  machinegun  (M240),  and  many  more.  The  ISMT  located  at  NPS 
can  train  up  to  two  individuals  at  a  time.  ISMT  has  the  unique  capability  to  “provide 
immediate  feedback  to  the  instructor  and  trainee  on  weapon  trigger  pull,  cant  position, 
barrel  movement,  rifle  butt  pressure,  tracing  of  the  muzzle  on  a  weapon  prior  to  and  after 
shoot,  and  grouping”  (Getty,  2008).  Immediate  feedback  is  vital  in  marksmanship 
training  because  it  provides  the  opportunity  for  participants  to  adjust  the  weapon 
accordingly,  thereby  greatly  improving  accuracy.  Just  as  important,  ISMT  records  muzzle 
movement  and  displays  the  recording  as  part  of  the  trace  profile  feature.  The  trace  profile 
feature  records  0.2  seconds  before  the  trigger  squeeze,  the  actual  trigger  squeeze,  and 
0.2  seconds  post  trigger  squeeze.  The  data  displayed  from  the  trace  profile  feature  allows 
the  instructor  to  critique  the  participant’s  technique,  which  would  be  nearly  impossible  to 
do  via  the  naked  eye. 

The  ISMT  lab  is  located  in  a  30-foot  trailer  on  the  NPS  campus.  The  front  section 
of  the  trailer  contains  the  stored  weapons  and  two  Windows-based  computers  that  run  the 
ISMT.  The  weapons  used  in  this  research  were  tethered  to  the  CO2  cylinder  hank  and  the 
ISMT  operating  system,  with  input  leads  consisting  of  a  hose  for  CO2  and  a  wire  harness 
to  provide  audio  and  system  feedback.  The  main  section  of  the  trailer  contains  the 
instructor  control  panel,  just  forward  of  the  firing  line.  This  was  used  to  start  and  stop 
each  scenario,  playback  scores,  and  switch  between  scenarios.  Also  behind  the  firing  line 
is  the  CO2  cylinder  bank  that  provides  the  recoil  for  the  weapons.  The  projector  that 
displays  the  video  for  the  scenario  and  the  camera  that  reads  the  laser  from  the  weapons 
are  mounted  on  the  ceiling  in  the  main  section  of  the  trailer.  The  large  projector  screen 
that  displays  the  scenarios  and  the  subwoofers  that  provide  audio  for  realistic  weapon 
firing  sound  effects  are  located  at  the  end  of  the  trailer. 
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D.  MEASURES 


1.  Demographics 

A  general  demographics  survey  was  administered  which  contained  questions 
about  such  topics  as  age,  service,  and  job  specialty  (see  Table  1). 

2.  Marksmanship  Experience 

A  survey  was  administered  that  asked  questions  regarding  participants’  previous 
marksmanship  experience  (see  Table  2). 

E.  ISMT  SCENARIOS 

The  ISMT  system  used  for  this  research  was  a  USMC  version  which  contained  a 
number  of  USMC-approved  scenarios,  as  well  as  some  user-made  Navy-specific 
scenarios,  including  the  Navy  Handgun  Qualification  Course  (NHQC).  The  two  ISMT 
scenarios  (low  resolution  and  high  resolution)  that  were  used  for  this  research  were 
designed  by  this  researcher  using  a  modified  version  of  the  NHQC. 

The  NHQC  was  conducted  as  follows:  a  total  of  thirteen  separate  Transtar  II 
silhouette  targets  were  displayed  one  at  a  time,  at  three  different  ranges  (see  Figure  1). 
Before  each  target  was  displayed  the  shooter  received  instructions  given  by  the  instructor. 
Each  target  appeared  for  a  set  duration  of  time,  and  had  to  be  shot  a  set  numbers  of  times 
to  receive  maximum  score. 
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Region  1  Outside  of  Target  Silhouette 


Figure  1.  Transtar  II  Silhouette  Target  with  Regions  (From  Getty,  2008) 

Normally  when  conducting  the  NHQC  on  a  live  fire  range,  the  shooter  will  be 
given  six  magazines  total;  four  magazines  with  six  rounds  in  each  magazine  for  the  three- 
and  seven-yard  line  firing  sequences,  and  two  magazines  with  12  rounds  in  each 
magazine  for  the  15 -yard  line  firing  sequence.  In  order  to  streamline  the  experiment, 
participants  were  given  a  single  magazine  which  they  simply  had  to  eject  and  reinsert  into 
the  weapon,  which  allowed  the  ISMT  software  to  recognize  the  weapon  as  being 
reloaded.  The  instructions  for  the  NHQC  firing  sequence  were  read  from  OPNAVINST 
3591.  IF  (Chief  of  Naval  Operations,  2009).  Refer  to  Table  3  for  a  detailed  description  of 
NHQC  firing  sequence,  Figure  2  for  the  participant’s  view  of  the  simulated  range,  and 

Figure  3  for  the  instructor’s  control  panel  display  screen. 
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Table  3.  Navy  Handgun  Qualification  Course  Firing  Sequence 


Yard  Line 

Rounds 

Sequence 

Remarks 

3 

12 

Draw  and  fire  2  rounds  in  4  seconds 

2  rounds  strong  hand  supported  position 

Draw  and  fire  2  rounds  in  4  seconds 

2  rounds  strong  hand  supported  position 

Draw  and  fire  2  rounds,  reload  6  rounds, 
and  fire  2  rounds  in  1 0  seconds 

4  rounds  strong  hand  supported  position 

Draw  and  fire  4  rounds  in  8  seconds 

2  rounds  strong  hand  supported  position, 

2  rounds  weak  hand  supported  position 

7 

12 

Draw  and  fire  2  rounds  in  4  seconds 

2  rounds  strong  hand  supported  position 

Draw  and  fire  2  rounds  in  4  seconds 

2  rounds  strong  hand  supported  position 

Draw  and  fire  2  rounds,  reload  6  rounds, 
and  fire  2  rounds  in  1 0  seconds 

4  rounds  strong  hand  supported  position 

Draw  and  fire  4  rounds  in  8  seconds 

2  rounds  strong  hand  supported  position, 

2  rounds  weak  hand  supported  position 

15 

24 

Draw  and  fire  2  rounds  in  4  seconds 

2  rounds  strong  hand  supported  position 

Draw  and  fire  2  rounds  in  4  seconds 

2  rounds  strong  hand  supported  position 

Draw  and  fire  4  rounds  in  8  seconds 

4  rounds  strong  hand  supported  position 

Draw  and  fire  4  rounds,  reload  12  rounds, 
and  fire  4  rounds  in  20  seconds 

8  rounds  strong  hand  supported  position 

Draw  and  fire  8  rounds  in  20  seconds 

8  rounds  kneeling  position 

Figure  3.  Instructor  Control  Panel 


1.  Low  Resolution  Scenario 

The  low  resolution  scenario  was  a  modified  version  of  the  NHQC  already 
programmed  into  the  ISMT  system.  The  NHQC  featured  a  background  image  of  a  U.S. 
Navy  warship  flight  deck,  an  image  which  would  be  familiar  to  anyone  in  the  Navy  who 
has  conducted  a  gun  qualification  on  the  flight  deck  of  a  ship.  This  scenario  was 
modified,  and  the  background  image  was  changed  to  a  plain  gray  screen.  The  time  delay 
between  each  target  was  also  reduced  in  order  to  minimize  the  time  required  by 
participants.  Other  than  these  two  changes,  the  scenario  followed  the  rest  of  the  NHQC. 
Figures  4  shows  the  low  resolution  scenario  targets  displayed  at  three,  seven,  and  15 
yards,  respectively. 
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Figure  4.  Low  Resolution  Scenario  Targets 


2.  High  Resolution  Scenario 

The  high  resolution  scenario  was  a  modified  version  of  the  NHQC  already 
programmed  into  the  ISMT  system.  The  NHQC  featured  a  background  image  of  a  U.S. 
Navy  warship  flight  deck,  an  image  which  would  be  familiar  to  anyone  in  the  Navy  who 
has  conducted  a  gun  qualification  on  the  flight  deck  of  a  ship.  This  scenario  was 
modified,  and  the  background  image  was  changed  to  an  image  of  a  hallway  inside  a 
building.  The  time  delay  between  each  target  was  also  reduced  in  order  to  minimize  the 
time  required  by  participants.  Other  than  these  two  changes,  the  scenario  followed  the 
rest  of  the  NHQC.  Figures  5  shows  the  high  resolution  scenario  targets  displayed  at  three, 
seven,  and  15  yards,  respectively. 


Figure  5.  High  Resolution  Scenario  Targets 
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F. 


PROCEDURES 


Before  participants  arrived  for  a  session,  the  author  ensured  the  ISMT  was  ready. 
First,  a  ready  line  and  a  firing  line  were  established  in  accordance  with  OPNAVINST 
3591.  IF  (Chief  of  Naval  Operations,  2009)  and  marked  in  ISMT  by  placing  masking  tape 
on  the  deck  at  the  appropriate  distances  from  the  projector  screen.  General  safety  rules 
were  posted  next  to  the  firing  line  and  participants  were  instructed  to  read  them  prior  to 
commencing  training.  The  author  ensured  the  ISMT  system  was  calibrated  prior  to  each 
participant’s  arrival.  Following  system  calibration,  the  M9  was  registered  and  calibrated 
to  the  ISMT.  The  registration  process  consisted  of  the  instructor  firing  one  round  at  the 
screen  when  prompted  by  the  ISMT.  The  laser-reading  camera  would  read  the  individual 
pulse-coded  laser  for  that  particular  weapon  and  register  it  in  the  ISMT  system. 

All  participants  were  either  students  or  faculty  at  NPS.  Recruitment  was 
conducted  via  e-mail.  Participants  signed  up  for  individual  30-minute  time  blocks 
throughout  a  two  week  period.  When  the  participants  arrived  at  the  ISMT  they  were 
given  a  consent  form  to  read  and  sign.  After  giving  consent,  participants  were  asked  to 
fill  out  a  demographics  survey.  The  demographics  survey  contained  questions  about  the 
participants’  general  characteristics,  marksmanship  skill  level,  and  marksmanship 
experience. 

Once  the  survey  was  complete  the  participants  received  a  small  amount  of 
baseline  training.  Prior  to  beginning  the  baseline  training,  every  participant  received  a 
safety  brief  in  addition  to  an  overview  of  the  procedures  used  in  this  part  of  the  training. 
The  safety  brief  was  conducted  in  accordance  with  OPNAVINST  3591. IF  (Chief  of 
Naval  Operations,  2009)  to  ensure  all  safety  requirements  were  met.  Baseline  training 
ensured  that  every  participant  began  with  a  basic  level  of  knowledge  on  how  to  properly 
handle  and  fire  the  M9,  which  included  the  proper  use  of  the  M9  and  specific  weapon 
commands  for  the  experiment  (Getty,  2008).  Because  all  participants  had  different 
competency  levels,  baseline  training  varied  between  participants.  Some  required  little  to 
no  training  at  all,  while  others  required  significant  training  and  practice.  In  all  cases, 
training  and  practice  was  continued  until  the  participant  felt  comfortable  with  the  weapon 
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and  the  ISMT  scenario,  and  indicated  they  were  ready  to  begin  the  experiment.  Baseline 
training  was  conducted  using  the  same  gray  background  as  in  the  low  resolution  scenario 
described  above.  Table  3  provides  a  detailed  description  of  the  baseline  training  topics. 


Table  4.  Baseline  Training  Program  (From  Getty,  2008) 


Baseline  Training 

General  Range  Safety  Rules 
General  Safety  Rules 
Weapon  Conditions 
Weapon  Commands 
Pistol  Safety  Rules 
Ready  Line 
Firing  Line 
Range  Operations 
Training  Time  Out 
Loading  the  Pistol 
Making  the  Pistol  Ready 
Fire 

Cease  Fire 

Unload,  Showing  the  Pistol  Clear 
Dry  Reload 
Isosceles  only 
Isosceles  Standing  Position 
Isosceles  High  Kneeling  Position 


The  NHQC  firing  sequence  was  used  for  all  data  collection  shooting  events.  Each 
participant  shot  from  the  same  location  marked  by  blue  masking  tape  on  the  floor  of  the 
ISMT  trailer  to  indicate  the  center  line  to  help  participants  center  their  bodies  on  the 
target.  Instead  of  using  holsters,  all  participants  were  instructed  to  hold  the  weapon  in  the 
low  ready  firing  position  between  each  target.  The  low  ready  position  consists  of  pointing 
the  weapon  downward  at  a  45  degree  angle,  with  the  safety  engaged,  and  the  trigger 
finger  straight  along  the  frame  of  the  weapon  (see  Figure  6).  All  instructions  were  read 
from  OPNAVINST  3591.  IF  (Chief  of  Naval  Operations,  2009)  to  ensure  standardization 
and  consistency.  All  participants  used  the  isosceles  standing  position  for  the  first 
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36  rounds  fired  and  the  isosceles  kneeling  position  for  the  last  12  rounds  fired  to  mitigate 
any  confounds  associated  with  various  firing  positions  and  maximize  consistency  (see 
Figures  7  and  8). 


Figure  6.  Low  Ready  Position 
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Figure  8.  Isosceles  High  Kneeling  Position  (From  Headquarters,  United  States 

Marine  Corps,  2003) 

After  baseline  training  and  practice  was  complete,  the  participants  began  the 
actual  experiment.  They  were  randomly  presented  with  either  the  low  resolution  scenario 
or  the  high  resolution  scenario  first.  Once  the  scenario  was  started,  it  advanced 
automatically  until  all  thirteen  targets  had  been  displayed  and  shot  at.  Instructions  were 
given  by  the  instructor  before  each  target,  prompting  the  shooter  on  how  long  the  target 
would  be  displayed,  and  how  many  times  they  needed  to  shoot  it.  The  only  other  input 
from  the  instructor  during  the  experiment  was  corrective  remarks,  for  example  the  remark 
“safety”  if  the  shooter  failed  to  disengage  the  safety  on  the  weapon.  No  other  assistance 
was  provided.  The  only  feedback  given  during  the  scenario  was  from  simulated  bullet 
holes  appearing  in  the  target  on  the  screen. 

At  the  conclusion  of  the  first  scenario  the  participants  were  asked  to  move  to  the 

forward  part  of  the  trailer  and  look  away  from  the  screen.  This  allowed  the  researcher  to 
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display  and  write  down  the  participants’  score,  and  then  load  the  second  scenario. 
Participants  were  not  told  how  well  they  scored  on  the  first  scenario  until  the  end  of  the 
experiment.  The  second  scenario  was  whichever  one  (high  resolution  or  low  resolution) 
the  participant  did  not  have  for  their  first  scenario.  After  loading  the  second  scenario  the 
participants  were  instructed  to  return  to  the  firing  line.  The  second  scenario  was  carried 
out  in  exactly  the  same  way  as  the  first  scenario,  with  all  the  same  commands. 

At  the  conclusion  of  the  second  scenario  the  participants  were  asked  to  fill  out  a 
short  post-study  survey.  The  survey  included  questions  about  which  scenario  the 
participants  felt  was  more  realistic,  which  scenario  they  believed  they  performed  better 
on,  and  whether  they  felt  graphics  quality  affects  realism  and  training.  There  were  also  a 
few  short-answer  questions  where  participants  could  provide  additional  feedback  and 
ideas  for  improvement.  Once  the  participants  were  finished  with  the  survey  the  author 
reviewed  their  answers,  and  asked  questions  about  some  of  them  to  gain  a  better 
understanding  as  to  what  the  participants  experienced.  Finally,  the  participants  were  told 
their  score  from  the  first  round,  and  showed  a  replay  of  their  second  round,  ending  with 
their  second  round  score.  Figure  9  shows  an  example  of  the  final  score  screen. 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 
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IV.  RESULTS 


A.  DATA  PREPARATION  AND  STATISTICAL  METHODS 

Demographics  survey  data  was  transcribed  from  paper  surveys  and  recorded  in  an 
Excel  spreadsheet.  It  was  analyzed  using  Pearson  Product  Moment  Correlation  and  two- 
sample  /-tests  with  the  data  analysis  tool  package  in  Excel  2007. 

All  experimental  data  sets  from  ISMT  were  transcribed  from  computer-generated 
data  screens  into  an  Excel  spreadsheet.  Experimental  data  sets  were  analyzed  using 
Pearson  Product  Moment  Correlation  and  two-sample  /-tests  with  the  data  analysis  tool 
package  in  Excel  2007. 

The  main  variable  for  hypothesis  testing  was  NHQC  score.  Scores  were  graded  in 
accordance  with  OPNAVINST  3591. IF  (Chief  of  Naval  Operations,  2009),  based  on 
each  shooter’s  performance,  with  higher  scores  indicating  better  performance. 

For  the  averaged  NHQC  score,  difference  scores  were  calculated  on  the  following 
data  sets:  scenario  one  score  to  scenario  two  score,  and  low  resolution-first  to  high 
resolution- first  score.  The  difference  scores  were  used  to  assess  changes  in  performance 
over  time  to  detennine  whether  any  significant  improvement  or  degradation  occurred 
within  each  group.  These  difference  scores  were  analyzed  using  two-sample  /-tests  to 
determine  whether  or  not  the  low  resolution-first  group  showed  greater  levels  of 
improvement  than  the  high  resolution- first  group. 

B.  OBJECTIVE  ANALYSIS 

1.  Scores 

A  total  of  17  NPS  students  and  faculty  participated  in  the  experiment.  Of  these, 
nine  received  the  low  resolution  scenario  first,  and  eight  received  the  high  resolution 
scenario  first.  The  average  score  for  all  17  low  resolution  scenarios  (213.24  ±  22.38)  and 
all  17  high  resolution  scenarios  (212.65  ±  22.62)  showed  no  significant  difference  ip  = 
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0.94).  Since  this  finding  partially  supports  HO2,  other  comparisons  were  examined  to  see 
if  graphics  quality  had  any  effect  on  score. 

The  first  thing  examined  was  whether  those  participants  who  received  the  high 
resolution  scenario  second  had  a  greater  improvement  in  their  scores  than  those  who 
received  the  low  resolution  scenario  second.  The  average  scores  for  scenario  one  (207.71 
±  23.84)  and  scenario  two  (218.18  ±  19.65),  clearly  show  that  the  participants  tended  to 
improve  from  their  first  scenario  to  their  second  scenario  (average  improvement  =  10.47 
±  10.24  points),  regardless  of  resolution.  In  fact,  all  but  two  participants  showed 
improvement  from  scenario  one  to  scenario  two.  This  “training  effect”  was  expected,  and 
was  the  reason  why  half  the  participants  received  the  low  resolution  scenario  second,  in 
order  to  try  and  counter  its  effects.  Further  decomposing  these  results  down  between 
groups,  those  participants  who  received  the  high  resolution  scenario  second  had  an 
average  improvement  of  9.33  ±  12.45  points,  while  those  who  received  the  low  resolution 
scenario  second  had  an  average  improvement  of  11.75  ±  7.67  points.  Once  again  these 
results  showed  no  significant  difference  (p= 0.63). 

Based  on  the  two  tests  above  it  appears  that  graphics  quality  does  not  have  any 
effect  on  average  score  or  score  improvement.  Therefore,  other  demographic  data  was 
examined  to  determine  if  there  were  other  factors  which  were  affecting  score. 

2.  Demographics 

The  first  demographic  looked  at  was  age.  Participants’  age  was  correlated  against 
their  average  score  using  Excel,  returning  a  value  of  0.27.  Using  a  degree  of  freedom  ( df ) 
of  15,  and  a  two-tailed  alpha  ( a )  of  0.05,  the  correlation  matrix  gives  us  a  significance 
level  of  0.482.  Therefore,  we  can  conclude  that  a  participants’  age  and  their  average  score 
are  not  correlated. 

The  next  demographic  looked  at  was  self-reported  currency  with  the  M9. 
Participants’  currency  was  correlated  against  their  average  score  using  Excel,  returning  a 
value  of  0.33.  Using  a  degree  of  freedom  ( df)  of  15,  and  a  two-tailed  alpha  (<7)  of  0.05, 
the  correlation  matrix  gives  us  a  significance  level  of  0.482.  Therefore,  we  can  conclude 

that  a  participants’  currency  and  their  average  score  are  not  correlated. 
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Another  demographic  looked  at  was  self-reported  proficiency  with  the  M9. 
Participants’  proficiency  was  correlated  against  their  average  score  using  Excel,  returning 
a  value  of  0.45.  Using  a  degree  of  freedom  ( df)  of  15,  and  a  two-tailed  alpha  (a)  of  0.05, 
the  correlation  matrix  gives  us  a  significance  level  of  0.482.  However,  since  it  is 
reasonable  to  assume  that  there  is  a  relationship  between  proficiency  and  average  score, 
using  a  one-tailed  a  of  0.05  instead  is  appropriate,  which  produced  a  significance  level  of 
0.412.  Based  on  this  we  can  conclude  that  a  participants’  proficiency  and  their  average 
score  are  correlated. 

The  author  also  investigated  whether  corrective  eyewear  had  an  effect  on 
participants’  average  score.  There  were  seven  participants  who  reported  wearing  glasses 
or  contact  lenses,  and  10  participants  who  did  not.  The  average  score  of  participants  with 
corrective  lenses  was  200.57  ±  25.00,  while  the  average  score  of  participants  with  no 
corrective  lenses  was  221.60  ±  13.52.  A  two-tailed  /-test  with  an  alpha  ( a )  of  0.05 
revealed  a  /7-value  of  0.752.  A  /7-value  below  0.05  is  generally  considered  statistically 
significant,  while  one  of  0.05  or  greater  indicates  no  difference  between  the  groups. 
Therefore,  we  can  conclude  that  wearing  corrective  eyewear  had  no  significant  effect  on 
average  score.  This  is  further  supported  by  the  fact  that  participants  who  wore  corrective 
eyewear  had  an  average  currency  of  1.86  and  an  average  proficiency  of  2.43,  while  those 
who  did  not  wear  corrective  eyewear  had  an  average  currency  of  2.80  and  an  average 
proficiency  of  3.40.  In  other  words,  those  participants  who  did  not  wear  glasses  or 
contacts  tended  to  have  a  greater  currency  and  proficiency  with  the  M9.  Because  of  this, 
we  cannot  say  with  certainty  if  their  experience  or  their  lack  of  corrective  eyewear 
affected  their  score. 

The  last  demographic  investigated  was  whether  participants  who  reported  playing 
video  games  scored  higher  than  those  who  did  not.  There  were  eight  participants  who 
reported  playing  video  games,  and  nine  participants  who  did  not.  The  average  score  of 
participants  who  played  video  games  was  221.06  ±  14.63,  while  the  average  score  of 
participants  who  did  not  was  205.72  ±  24.29.  A  two-tailed  /-test  with  an  alpha  ( a )  of  0.05 
revealed  a  /7-value  of  0.1338.  A  P-value  below  0.05  is  generally  considered  statistically 
significant,  while  one  of  0.05  or  greater  indicates  no  difference  between  the  groups. 
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Therefore,  we  can  conclude  that  playing  video  games  had  no  significant  effect  on  average 
score.  This  is  further  supported  by  the  fact  that  participants  who  played  video  games  had 
an  average  currency  of  2.63  and  an  average  proficiency  of  3.13,  while  those  who  did  not 
play  video  games  had  an  average  currency  of  2.22  and  an  average  proficiency  of  2.89.  In 
other  words,  those  participants  who  played  video  games  tended  to  have  a  greater  currency 
and  proficiency  with  the  M9.  Because  of  this,  we  cannot  say  with  certainty  if  their 
experience  or  familiarity  with  video  games  affected  their  score. 

C.  SUBJECTIVE  ANALYSIS 

While  the  primary  goal  of  this  research  was  to  collect  objective  data  on 
participants’  scores,  a  large  amount  of  subjective  data  was  also  recorded.  This  came  in 
the  form  of  participants  answers  to  the  post-testing  survey  questions,  as  well  as  infonnal 
interviews  between  the  participants  and  the  author  following  each  experimental  session. 
Given  the  lack  of  significance  in  the  objective  data  collected,  it  is  this  subjective  data  that 
ends  up  being  the  most  interesting,  and  perhaps  the  most  useful  to  the  research.  Of  the  17 
participants  in  this  study,  all  17  completed  the  post-testing  survey. 

1.  Graphics  Quality  and  Realism 

As  part  of  the  post-testing  survey,  participants  were  asked  to  rate  the  realism  and 
graphics  quality  of  each  scenario  on  a  scale  of  one  to  five,  with  one  being  low  and  five 
being  high.  The  average  score  for  realism  of  the  low  resolution  scenario  was  2.12,  and  for 
the  high  resolution  scenario  it  was  3.06.  A  two-tailed  /-test  with  an  alpha  (a)  of  0.05 
revealed  a  /;- value  of  0.011,  meaning  there  was  a  statistically  significant  difference 
between  the  two  scores.  Similarly,  the  average  score  for  graphics  quality  of  the  low 
resolution  scenario  was  1.71,  and  for  the  high  resolution  scenario  it  was  2.71.  Again,  a 
two-tailed  /-test  with  an  alpha  (a)  of  0.05  was  conducted  and  returned  a  /7-value  of  0.001, 
meaning  there  was  a  statistically  significant  difference  between  these  two  scores  as  well. 

What  these  two  results  tell  us  is  that  regardless  of  their  performance,  almost  every 
single  participant  felt  that  the  high  resolution  scenario  had  better  graphics  quality  and  was 
more  realistic  than  the  low  resolution  scenario.  What  is  not  clear  from  the  survey  alone  is 
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why  participants  felt  this  way.  The  informal  interviews  conducted  after  the  testing  helped 
shine  some  light  on  this  issue.  One  factor  that  was  pointed  out  by  a  number  of 
participants  was  the  contrast  between  the  background  and  the  target.  They  felt  that  the 
target  stood  out  more  against  the  high  resolution  background,  allowing  them  to  more 
quickly  and  accurately  engage  it.  Another  factor  along  these  same  lines  was  feedback. 
When  the  target  is  shot,  simulated  bullet  holes  appear.  Because  the  low  resolution 
background  is  almost  the  same  color  as  the  center  portion  of  the  target,  these  bullet  holes 
are  difficult  to  see.  They  become  much  more  visible  on  the  high  resolution  background. 
Many  participants  believed  that  this  improved  feedback  and  made  the  simulation  more 
realistic.  Finally,  some  participants  felt  that  the  high  resolution  background  offered  a 
better  frame  of  reference.  On  the  plain  gray  background  of  the  low  resolution  scenario, 
participants  were  unsure  where  the  target  was  going  to  appear.  Conversely,  the  image  of 
the  hallway  in  the  high  resolution  scenario  gave  them  a  frame  of  reference  as  to  where  to 
expect  the  target  to  appear;  in  this  case,  in  the  center  of  the  hallway. 

2.  Perceived  Performance 

One  question  on  the  post-testing  survey  asked  participants  to  pick  which  scenario 
they  believed  they  performed  better  on  (first  or  second).  Participants  answered  this 
question,  along  with  the  rest  of  the  post-testing  survey,  before  they  were  given  their 
scores  from  either  scenario.  Of  the  17  participants  in  this  study,  10  believed  they 
performed  better  on  the  second  scenario,  while  seven  believed  they  performed  better  on 
the  first  scenario.  This  result  was  quite  shocking,  as  it  was  expected  most  participants 
would  feel  they  did  better  on  the  second  scenario,  simply  due  to  having  more  practice 
from  the  first  scenario.  Participants  were  asked  why  they  felt  they  did  better  on  a 
particular  scenario,  and  the  answers  given  were  quite  interesting.  While  a  large  portion  of 
them  felt  they  performed  better  on  the  second  scenario  solely  due  to  practice,  a  number  of 
other  reasons  were  given  as  well.  At  least  three  participants  felt  they  did  better  on  the  first 
scenario  because  they  knew  they  missed  a  shot  on  the  second  scenario,  and  therefore 
believed  that  their  first  scenario  score  was  better.  In  these  cases  it  did  not  matter  if  they 
had  the  low  resolution  or  high  resolution  scenario  first. 
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Another  three  participants  felt  that  they  performed  better  on  the  first  scenario 
because  it  was  less  distracting.  In  all  three  cases  they  had  the  low  resolution  scenario 
first,  followed  by  the  high  resolution  scenario.  Interestingly  enough,  all  three  still  rated 
the  high  resolution  scenario  as  more  realistic  and  having  better  graphics  quality.  As  one 
participant  explained  it,  if  they  wanted  to  shoot  solely  for  the  purpose  of  getting  a  high 
numerical  score,  they  would  prefer  the  low  resolution  scenario,  but  if  they  wanted  to 
receive  meaningful,  realistic  training,  they  would  rather  shoot  the  high  resolution 
scenario. 

One  participant  felt  they  performed  better  on  the  first  scenario  (high  resolution  in 
this  case)  because  it  offered  a  better  spatial  frame  of  reference,  as  discussed  above.  There 
was  also  one  participant  who  felt  they  did  better  on  the  second  scenario  (high  resolution) 
because  of  the  improved  feedback.  Finally,  there  was  one  participant  who  felt  they  did 
better  on  the  second  scenario  (high  resolution)  because  it  didn’t  have  the  “glairing  white 
background”  of  the  first  scenario. 

In  summary,  while  many  participants  felt  they  improved  with  practice,  regardless 
of  background  type,  there  were  some  who  preferred  the  low  resolution  scenario  because  it 
was  less  distracting,  and  some  who  preferred  the  high  resolution  scenario  because  it  was 
less  glairing  and  offered  a  spatial  frame  of  reference  and  better  visual  feedback. 
Interesting  enough,  no  participant  explicitly  cited  graphics  quality  or  realism  as  reasons 
why  they  felt  they  performed  better. 

3.  How  Graphics  Quality  Affects  Realism  and  Training 

As  part  of  the  post-testing  survey,  participants  were  asked  if  they  agreed  with 
three  statements  on  a  scale  of  one  to  five,  with  one  being  “strongly  disagree”  and  five 
being  “strongly  agree.”  The  three  statements  were:  “Graphic  quality  affects  realism,” 
“Improving  the  graphics  quality  would  improve  the  realism  of  the  simulation,”  and 
“Improving  the  graphics  quality  would  improve  the  training  received.”  The  average 
responses  for  these  three  statements  were  4.35,  4.41,  and  4.29,  respectively.  The  lowest 
response  given  to  a  question  was  a  three  (or  neutral),  and  only  three  of  these  were  given 
total.  All  the  remaining  answers  were  either  a  4  (agree)  or  a  5  (strongly  agree). 
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These  results  are  likely  the  most  important  of  the  entire  study.  They  show  that  a 
diverse  group  of  military  personnel,  with  different  backgrounds  and  experience  levels,  all 
unanimously  believe  that  improving  the  graphics  quality  of  the  simulator  would  improve 
the  training  received.  This  should  serve  as  strong  incentive  for  the  Department  of  the 
Navy,  and  the  Department  of  Defense  as  a  whole,  to  study  this  issue  further  before 
procurements  of  new  simulators  is  made. 

4.  Open  Answers  and  Additional  Feedback 

The  final  part  of  the  post-testing  survey  asked  the  participants  to  provide 
additional  feedback  via  three  open- answer  type  questions.  These  questions  were: 

“What  other  factors  besides  graphics  quality  do  you  believe  affects  realism?” 

“What  factors  would  you  change  to  improve  the  training  simulation?” 

“Do  you  have  any  other  ideas  for  improvement?” 

A  wide  array  of  answers  were  given,  including  sound  effects,  voice,  target 
movement,  general  ambience  of  the  training  environment,  interactive  scenarios,  wearing 
combat  gear,  smoke,  requiring  the  shooter  to  move,  lighting,  and  quality  of  the  weapon 
analogue. 

A  number  of  participants  believed  that  additional  sounds,  background  noise,  and 
distractions  in  general  would  improve  the  realism  of  the  simulator.  Things  like  people 
talking,  vehicle  noises,  and  even  other  weapons  being  fired  were  cited  as  examples. 

Many  participants  believed  that  having  better,  more  realistic  scenarios  would 
greatly  improve  the  overall  realism  of  the  simulator.  Some  commonly  cited  examples 
were:  have  targets  that  look  like  real  human  beings,  have  targets  that  move,  have  targets 
that  speak,  make  gestures,  and  points  weapons  (a  way  to  show  hostile  intent),  require  the 
shooter  to  interact,  make  decisions,  and  even  issue  voice  commands. 

Environment  and  ambiance  were  considered  important  factors  as  well.  Some 
participants  found  the  lighting  in  the  trailer  (dark  trailer  with  bright  screen)  to  be  highly 
unrealistic,  and  believed  that  a  more  natural  daylight  or  nighttime  system  would  be  better. 
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Other  environmental  factors  such  as  wind,  rain,  hot  and  cold  temperatures,  and  even 
smells  were  cited  as  possible  ideas  for  improvement. 

Finally,  a  number  of  the  participants  commented  on  the  quality  of  the  weapon 
analogue.  While  most  participants  found  it  highly  realistic,  a  number  of  problems  were 
pointed  out.  First,  the  air  house/data  cable  tether  that  connects  the  weapon  to  the  rest  of 
the  ISMT  makes  movement  more  restricted,  and  causes  a  break  in  the  users’  immersion 
in  the  simulator.  It  also  makes  the  weapon  harder  to  reload,  which  could  adversely  affect 
training.  Second,  while  the  air  system  allows  the  weapon  to  recoil,  it  has  nowhere  near 
the  same  force  as  a  real  live-fire  weapon.  Similarly,  the  sound  of  the  weapon  firing  comes 
from  the  simulator  speakers,  and  is  a  fraction  of  the  loudness  of  a  true  firearm.  Finally,  no 
muzzle  flash  is  produced  when  the  weapon  analogue  is  fired.  All  these  factors  hamper  the 
realism  of  an  otherwise  outstanding  weapon  analogue. 
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V.  CONCLUSION  AND  RECOMMENDATIONS 


A.  SUMMARY 

The  purpose  of  this  research  was  to  determine  what  factors  impact  the  realism  of 
virtual  marksmanship  simulators,  identify  how  these  factors  can  be  improved,  and 
measure  the  resulting  impact  on  trained  performance  In  the  future  this  will  allow  us  to 
enhance  the  realism  of  these  factors,  which  will  in  turn  enhance  the  overall  realism  of  the 
simulators.  This  is  important  because  being  able  to  clearly  define  what  makes  these 
simulators  more  realistic  will  lead  to  further  research  to  determine  if  simulators  with 
enhanced  realism  are  more  effective  at  training  than  their  less  realistic  counterparts. 

This  study  focused  primarily  on  the  factor  of  graphics  quality  of  the  ISMT.  The 
goal  was  to  see  if  improving  the  graphics  quality  of  an  ISMT  scenario  (by  changing  the 
background  image)  would  have  any  impact  on  the  perfonnance  of  the  shooters,  or  affect 
their  subjective  experience  of  the  simulator. 

B.  HYPOTHESIS  ONE  DISCUSSION 

Are  there  specific  factors  which  impact  the  realism  of  virtual  marksmanship 
simulators? 

HOi:  No  specific  factors  can  be  identified  which  impact  the  realism  of  virtual 
marksmanship  simulators. 

HAi:  One  or  more  specific  factors  are  identified  which  impact  the  realism  of 
virtual  marksmanship  simulators. 

While  no  conclusive  objective  data  identified  any  factors  which  impact  the 
realism  of  virtual  marksmanship  simulators,  there  was  subjective  data  to  suggest  that 
graphics  quality,  as  well  as  a  number  of  other  factors,  impact  the  realism  of  virtual 
marksmanship  simulators. 

Based  on  their  answers  to  the  post-testing  survey,  participants  overwhelmingly 

agreed  that  graphics  quality  affects  realism,  with  only  a  single  participant  being  neutral. 
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Participants  also  provided  a  wealth  of  other  possible  factors  which  may  affect  realism. 
These  included  sound  effects,  voice,  target  movement,  general  ambience  of  the  training 
environment,  interactive  scenarios,  wearing  combat  gear,  smoke,  requiring  the  shooter  to 
move,  lighting,  and  quality  of  the  weapon  analogue. 

C.  HYPOTHESIS  TWO  DISCUSSION 

Does  improving  or  enhancing  these  factors  enhance  the  overall  simulation? 

HO  2:  Improving  or  enhancing  a  factor,  such  as  graphics  quality,  will  have  no 
discernible  effect  on  a  participants  objective  score  or  their  subjective  experience  with  the 
simulator. 

HA2:  Improving  or  enhancing  a  factor,  such  as  graphics  quality,  will  have  a 
positive  effect  on  a  participants  objective  score  and/or  their  subjective  experience  with 
the  simulator. 

While  the  objective  data  taken  from  shooters  scores  showed  no  improvement 
between  low  resolution  and  high  resolution  scenarios,  the  subjective  experience  of  the 
participants  showed  positive  results.  There  was  a  significant  difference  between  both  the 
graphics  quality  and  realism  scores  reported  on  the  participants’  post-testing  surveys. 
Similarly,  participants  overwhelmingly  agreed  that  graphics  quality  affects  realism,  with 
only  a  single  participant  being  neutral.  Finally,  during  the  informal  interviews  the 
majority  of  participants  stated  that  they  preferred  the  high  resolution  scenario,  and  felt  it 
provided  better  and  more  realistic  training. 

D.  FUTURE  WORK  AND  RECOMMENDATIONS 

The  limited  size  and  scope  of  this  experiment  limited  the  quality  of  the  results 
obtained.  Nevertheless,  it  provided  some  useful  insight  into  possible  future  studies,  and 
possible  ways  to  improve  virtual  marksmanship  simulators. 

The  first  thing  to  try  would  be  varying  aspects  other  than  just  the  background 
image.  Using  more  complex  images,  different  targets,  moving  objects,  and  other  changes 
could  easily  be  made  to  the  existing  ISMT  program  and  tested  in  the  same  manner. 
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Likewise,  a  background  image  could  be  made  with  true  high  resolution  and  low 
resolution  versions,  for  example  640x480  pixels  vs.  1600x1200  pixels.  Comparing  these 
would  show  if  actual  screen  resolution  has  any  impact  on  performance. 

Another  avenue  for  future  study  would  be  to  investigate  the  other  training  modes 
present  in  ISMT.  Currently,  ISMT  has  three  training  modes:  Lanes  Training,  CGI 
Collective  Training,  and  Video  Training.  A  brief  description  of  these  three  modes 
follows. 

1.  Lanes  Training 

Lanes  Training  is  the  most  basic  training  mode  offered  in  ISMT.  It  allows  the  user 
to  replicate  the  look  and  feel  of  a  standard  target  range.  Different  background  images  can 
be  applied  to  mimic  everything  from  being  indoors  in  a  building,  to  being  outside  in  the 
woods,  to  being  on  the  flight  deck  of  a  ship.  However,  in  all  cases  the  backgrounds  are 
two-dimensional  images  and  only  one  can  be  applied  per  scenario.  The  point  of  view  is 
also  fixed,  so  the  user  cannot  “look  around.”  A  variety  of  targets  can  be  added  to  the 
scenario,  ranging  from  simple  silhouettes  to  images  of  real  people.  While  the  targets  can 
be  made  to  move  and  “pop  up,”  they  are  still  all  two-dimensional  images,  which  limits 
the  complexity  and  realism  of  the  scenarios  possible.  This  experiment  used  Lanes 
Training  as  the  sole  mode  for  data  collection. 

2.  Video  Training 

Video  Training  is  a  slightly  more  advanced  mode  of  training  than  Lanes  Training, 
with  a  number  of  additional  features.  Video  Training  uses  full  motion  video  (FMV)  in 
place  of  a  background  image.  In  this  mode,  a  video  will  play,  recorded  in  the  real  world 
by  real  actors.  As  the  video  plays  the  actors  will  move  about,  talk,  and  interact  with  each 
other.  At  some  point  during  the  video  an  actor  who  is  an  enemy  target  may  appear,  at 
which  point  the  shooters  will  have  to  engage  the  target.  This  mode  offers  a  number  of 
benefits  over  Lanes  Training,  the  first  of  which  is  the  obvious  increase  in  realism 
afforded  by  having  a  live-action  video  playing,  with  real,  interactive  targets.  This  mode 
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can  also  be  used  to  train  decision  making  skills,  as  the  shooters  have  to  make  on-the-spot 
judgment  calls  as  to  whether  a  target  is  displaying  hostile  intent  or  not,  and  whether  or 
not  they  should  engage. 

Despite  these  benefits,  Video  Training  still  has  a  number  of  limitations.  First,  it  is 
not  nearly  as  easy  to  modify  as  Lanes  Training.  In  Lanes  Training,  creating  custom  user 
scenarios  is  quite  simple  and  straight  forward,  as  only  the  background  and  targets  need  to 
be  manipulated.  Creating  a  custom  Video  Training  scenario  would  require  filming  one  or 
more  new  videos,  editing  them,  importing  them  into  the  ISMT  system,  and  then 
overlaying  them  with  all  the  appropriate  target  parameters.  The  reality  is  most  users  will 
never  do  this,  therefore  the  same  set  of  premade  Video  Training  scenarios  will  be  used 
over  and  over  again,  until  the  shooters  start  to  memorize  where  the  targets  appear,  at 
which  point  the  training  becomes  less  effective. 

As  with  all  the  ISMT  technology,  Video  Training  is  quite  old.  The  quality  of  the 
videos  is  relatively  low,  and  it  can  often  be  difficult  to  make  out  friend  or  foe.  The 
inherent  limitations  of  using  FMV  as  a  background  source  also  adversely  affect  realism. 
This  is  because  multiple  videos  have  to  be  filmed  for  each  scenario  to  account  for 
multiple  branching  paths.  However,  because  there  are  infinite  possible  paths,  and  only  a 
finite  number  of  videos  can  be  filmed,  the  scenario  is  often  left  with  gaps.  Imagine  the 
following  scenario: 

You  are  presented  a  video  where  you  are  standing  guard  duty  at  the  gate  to 
a  base.  A  person  enters  the  video  from  the  left  side  of  the  screen  and 
begins  to  walk  towards  you.  Suddenly  they  pull  out  a  weapon  and  begin 
firing  at  you  as  they  run  towards  the  right  side  of  the  screen. 

In  this  case,  the  producers  of  the  video  have  only  recorded  two  possible  branches. 
If  you  fail  to  engage  the  target,  or  miss  the  target,  he  successfully  runs  away.  If  you 
accurately  engage  the  target,  the  person  collapses  at  the  right  edge  of  the  screen.  But  what 
if  you  shot  the  target  as  soon  as  he  pulled  out  his  weapon?  Since  he  continues  to  move 
and  fire  at  you,  you  might  logically  conclude  that  you  missed  and  keep  shooting  him. 
Only  once  he  finally  collapses  at  the  edge  of  the  screen  would  you  receive  the  necessary 
feedback  to  know  that  you  successfully  engaged  the  target. 
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This  mode  could  be  investigated  in  a  number  of  ways  to  further  the  research.  A 
comparison  could  be  done  between  simple  moving  targets  in  Lanes  Training,  and  live- 
action  moving  actors  in  Video  Training.  In  addition,  as  with  the  Lanes  Training,  true  high 
resolution  and  low  resolution  versions  of  the  same  video  could  be  compared  against  each 
other  to  see  if  there  is  any  difference  in  performance. 

3.  CGI  Collective  Training 

The  final  mode  built  in  to  the  ISMT  is  the  CGI  (Computer-generated  imagery) 
Collective  Training  mode.  It  is  by  far  the  most  advanced  mode  available,  and  it  closely 
resembles  a  commercial  video  game.  CGI  Collective  Training  is  also  the  only  three- 
dimensional  mode  available.  In  this  mode,  full  3D  environments  can  be  built  through  the 
use  of  an  editor.  Then  3D  models  of  everything  from  people  to  vehicles  to  aircraft  can  be 
inserted  into  this  environment.  The  shooter  is  shown  the  field  of  view  of  his  avatar.  The 
avatar  can  be  moved  throughout  the  environment,  and  change  its  field  of  view  using  a 
joystick  connected  to  the  instructor  terminal.  The  avatar  can  also  be  mounted  in  a  vehicle, 
and  the  vehicle  can  be  programmed  to  follow  a  certain  course  through  the  environment, 
simulating  a  convoy  operation. 

The  benefits  of  this  mode  over  the  previous  two  are  quite  obvious.  Having  a  3D 
environment  allows  for  much  greater  immersion  on  the  part  of  the  shooter,  and  it  also 
allows  for  much  more  complex  scenarios  to  be  carried  out.  Using  computer-generated  3D 
models  as  targets  also  eliminates  the  problems  encountered  with  the  Video  Training 
mentioned  above.  Imagine  the  same  scenario;  now  when  the  target  is  shot  it  will  collapse 
immediately,  wherever  it  is,  providing  the  correct  feedback  to  the  shooter,  which  should 
result  in  better  training. 

As  with  all  the  ISMT  technology,  CGI  Collective  Training  is  quite  old.  The 
quality  of  the  3D  models  and  environments  is  relatively  low,  and  it  can  often  be  difficult 
to  make  out  friend  or  foe.  Most  modern  video  games  have  exponentially  better  video 
quality,  and  as  such  young  trainees  who  are  familiar  with  video  games  may  find  the 
graphics  of  the  CGI  Collective  Training  to  be  unremarkable. 
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The  author  believes  that  this  mode  holds  the  most  potential  for  future  research. 
The  panicle  of  this  research  would  be  to  find  some  way  to  use  a  modern  video  game 
graphics  engine,  such  as  the  ones  found  in  popular  games  like  Call  of  Duty,  Crysis,  or 
Halo,  in  the  ISMT.  This  could  then  be  compared  against  the  standard  CGI  Collective 
Training  scenario,  to  see  if  shooter  perfonnance  improves. 

E.  CONCLUSION 

The  results  of  this  research  indicate  that:  (1)  changing  the  background  image  in 
the  Lanes  Training  mode  of  the  ISMT  has  no  affect  on  shooter  perfonnance,  (2)  the  vast 
majority  of  participants  believe  that  increasing  the  graphics  quality  of  ISMT  would 
increase  the  realism  of  the  simulation,  and  increase  the  training  received,  and  (3)  the 
majority  of  participants  believed  that  there  were  many  other  factors  which  could  be 
introduced  or  increased  in  order  to  make  the  simulation  more  realistic,  and  improve  the 
overall  training  experience.  This  study  has  only  touched  the  surface  of  what  is  possible. 
There  are  a  number  of  excellent  future  research  opportunities  available,  from  testing 
different  modes  of  ISMT,  to  varying  some  other  factors  that  might  affect  realism  of  the 
simulation.  Given  the  state  of  the  economy,  growing  DoD  budget  cuts,  and  an  increasing 
shift  to  a  high-tech  world,  simulation  training  is  becoming  more  important  than  ever 
before.  Finding  effective  ways  to  improve  these  simulations  so  they  provide  the  best, 
most  realistic  training  possible  will  be  critical  in  making  sure  our  armed  forces  are 
prepared  for  future  conflicts. 
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APPENDEX  A.  INFORMED  CONSENT  FORM 


Naval  Postgraduate  School 
Consent  to  Participate  in  Research 

Introduction.  You  are  invited  to  participate  in  a  research  study  entitled  Increasing  Realism  in 
Virtual  Marksmanship  Simulators.  The  purpose  of  the  research  is  to  determine  what  factors  impact 
the  realism  of  virtual  marksmanship  simulators,  identify  how  these  factors  can  be  improved,  and 
measure  the  resulting  impact  on  trained  performance. 

Procedures.  Subjects  will  be  asked  to  provide  consent  and  answer  an  initial  questionnaire.  These 
will  be  done  when  the  subject  first  enters  the  simulator,  and  should  take  approximately  15  minutes 
to  complete.  Next  the  subject  will  be  given  a  safety  brief,  and  then  trained  in  the  use  of  the 
simulator,  including  how  to  load,  fire,  and  operate  the  simulator  weapons.  This  process  will 
conclude  with  the  subject  calibrating  the  weapon  to  their  own  unique  shooting  profile.  The  duration 
for  the  safety  brief,  training,  and  calibration  is  approximately  15-20  minutes.  As  the  experiment 
begins,  the  subject  will  be  presented  with  two  to  four  different  scenarios,  some  at  a  “high 
resolution”  setting,  and  some  at  a  “low  resolution”  setting.  Subjects  will  be  required  to  fire  at 
various  targets  throughout  these  scenarios.  Their  scores  will  be  recorded.  Time  to  complete  the 
scenarios  is  approximately  10-15  minutes.  Finally  the  subjects  will  be  presented  with  a  follow-up 
survey,  where  they  will  be  asked  various  questions  about  the  scenarios  and  their  performance.  The 
survey  should  take  approximately  1 0  minutes. 

Location.  The  interview/survey/experiment  will  take  place  in  the  ISMT  Simulator,  located  behind 
Watkins  Hall,  on  the  NPS  campus. 

Cost.  There  is  no  cost  to  participate  in  this  research  study. 

Voluntary  Nature  of  the  Study.  Your  participation  in  this  study  is  strictly  voluntary.  If  you  choose 
to  participate  you  can  change  your  mind  at  any  time  and  withdraw  from  the  study.  You  will  not  be 
penalized  in  any  way  or  lose  any  benefits  to  which  you  would  otherwise  be  entitled  if  you  choose 
not  to  participate  in  this  study  or  to  withdraw.  The  alternative  to  participating  in  the  research  is  to 
not  participate  in  the  research. 

Potential  Risks  and  Discomforts.  The  potential  risks  of  participating  in  this  study  are:  Simulator 
sickness,  eye  strain,  and  arm  strain  from  holding  the  weapon.  You  will  be  notified  about  all  the 
risks  before  beginning  the  experiment,  and  reminded  that  you  can  and  should  stop  as  soon  as  you 
feel  any  discomfort.  The  researchers  will  constantly  monitor  for  any  signs  of  discomfort,  and 
terminate  the  experiment  if  they  believe  you  are  experiencing  any  discomfort.  Y ou  will  be  given 
time  to  rest  between  scenarios  in  order  to  minimize  eye  strain  and  arm  fatigue. 

Anticipated  Benefits.  Anticipated  benefits  from  this  study  are  the  potential  to  help  define  what 
factors  affect  realism  in  virtual  marksmanship  simulators,  and  determine  if  improving  these  factors 
improves  the  overall  simulation.  This  knowledge  can  be  used  to  help  create  new,  more  realistic, 
more  effective  simulators  in  the  future.  You  will  not  directly  benefit  from  your  participation  in  this 
research. 
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Compensation  for  Participation.  No  tangible  compensation  will  be  given. 

Confidentiality  &  Privacy  Act.  Any  information  that  is  obtained  during  this  study  will  be  kept 
confidential  to  the  full  extent  permitted  by  law.  All  efforts,  within  reason,  will  be  made  to  keep 
your  personal  information  in  your  research  record  confidential  but  total  confidentiality  cannot  be 
guaranteed.  All  hard-copy  records  will  be  kept  in  a  folder,  which  will  always  be  in  the  possession 
of  the  researcher,  or  locked  in  a  secure  location.  All  electronic  data  will  be  stored  in  a  secure 
folder  on  the  researchers  laptop. 

Points  of  Contact.  If  you  have  any  questions  or  comments  about  the  research,  or  you  experience  an 
injury  or  have  questions  about  any  discomforts  that  you  experience  while  taking  part  in  this  study 
please  contact  the  Principal  Investigator,  Dr.  Rudy  Darken,  831-656-7588,  darken@nps.edu. 
Questions  about  your  rights  as  a  research  subject  or  any  other  concerns  may  be  addressed  to  the 
Navy  Postgraduate  School  IRB  Chair,  CAPT  John  Schmidt,  USN,  831-656-3864, 
jkschmid@nps.edu. 

Statement  of  Consent.  I  have  read  the  information  provided  above.  I  have  been  given  the 
opportunity  to  ask  questions  and  all  the  questions  have  been  answered  to  my  satisfaction.  I  have 
been  provided  a  copy  of  this  form  for  my  records  and  I  agree  to  participate  in  this  study.  I 
understand  that  by  agreeing  to  participate  in  this  research  and  signing  this  form,  I  do  not  waive 
any  of  my  legal  rights. 


Participant’s  Signature 


Date 


Researcher’s  Signature 


Date 
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APPENDEX  B:  BEFORE  TESTING  SURVEY 


BEFORE  TESTING: 

1.  Service: _ 

2.  MOS  /  Designator: _ 

3-  Age: _ 

4.  Do  you  wear  glasses  or  contacts?  YES  /  NO 

5.  Have  you  ever  qualified  with  the  M9?  YES  /  NO 


6.  How  would  you  rate  your  currency  with  the  M9? 

- 

- - 

- 

u 

very  out  of 

out  of  date 

neutral 

current 

very  current 

date 

7.  How  would  you  rate  your  proficiency  with  the  M9? 

u 

u 

“W* - 

— u 

complete 

beginner 

average 

proficient 

very 

novice 

proficient 

8.  Have  you  ever  used  a  Virtual  Marksmanship  Simulator?  YES  /  NO 

9.  Do  you  play  video  games?  YES  /  NO 

10.  About  how  many  hours  per  week  do  you  play  video  games? 

OOO - 0—0 

0-5  6-10  11-15  16-20  21+ 
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APPENDEX  C:  AFTER  TESTING  SURVEY 


AFTER  TESTING: 


1 .  How  would  you  rate  the  realism  of  the  first  scenario? 

0—0 - o - o- 

very  unrealistic  neutral  realistic 

unrealistic 

2.  How  would  you  rate  the  graphic  quality  of  the  first  scenario? 

0  0  0 - O- 

very  low  low  neutral  high 

3.  How  would  you  rate  the  realism  of  the  second  scenario? 

O - 0  o - o- 

very  unrealistic  neutral  realistic 

unrealistic 

4.  How  would  you  rate  the  graphic  quality  of  the  second  scenario? 

0  0  0 - O- 

very  low  low  neutral  high 


-O 

very 

realistic 


-o 

very  high 


-O 

very 

realistic 


-o 

very  high 


5.  On  which  scenario  would  you  say  you  performed  the  best? _ 

For  the  following  three  questions,  please  choose  whether  you  agree  or  disagree  with  the 
statement. 


6.  Graphics  quality  affects  realism. 

0—0 - 


o 


o 


o 


strongly 

disagree 


disagree 


neutral 


agree 


strongly 

agree 


7.  Improving  the  graphics  quality  would  improve  the  realism  of  the  simulation. 


imp 

O 


O 


O' 


O 


O 


strongly 

disagree 


disagree 


neutral 


agree 


strongly 

agree 


8.  Improving  the  graphics  quality  would  improve  the  trainingreceived. 


O 


O 


o 


o 


o 


strongly 

disagree 


disagree 


neutral 


agree 


strongly 

agree 
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9.  What  other  factors  besides  graphics  quality  do  you  believe  affect  realism? 


10.  What  factors  would  you  change  to  improve  the  training  simulation? 


1 1 .  Do  you  have  any  other  ideas  for  improvements? 
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INITIAL  DISTRIBUTION  LIST 


1.  Defense  Technical  Information  Center 
Ft.  Belvoir,  Virginia 

2.  Dudley  Knox  Library 
Naval  Postgraduate  School 
Monterey,  California 
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